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ABSTRACT 



The taxonomy presented in this paper lends structure to the 
range of tasks or problems possible within an interactive graphical medium. 
Some 70 items from a testing project were the basis for forming the 
categories described in the taxonomy. Items were drawn from science and 
technology domains. The categorical scheme was refined iteratively by two 
raters until it was able to accommodate all items. The framework is intended 
to have utility in the design of problems cast within an interactive 
graphical medium. The following item types are included: (1) point to image 

element; (2) evaluate image; (3) show direction/path; (4) show height, 
extent, or boundaries; (5) assemble elements; 6) indicate categorical, 
ordinal, or functional relationships; and (7) indicate continuous 
relationships. (Contains one table and eight references.) (Author/SLD) 
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Summary 

The taxonomy presented in this paper lends structure to the range of tasks or problems possible 
within an interactive graphical medium. Some 70 items from a testing project were the basis for 
forming the categories described in the taxonomy. Items were drawn from science and technology 
domains. The categorical scheme was refined iteratively by two raters until it was able to 
accommodate all items. The framework is intended to have utility in the design of problems cast 
within an interactive graphical medium. 
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An Initial Taxonomy of Methods 

for Demonstration of Proficiency in a Figural Medium 

Computer-based testing and learning environments afford the opportunity for users to interact with 
knowledge representations that can take a variety of forms. Among these, interactive graphical 
repwesentations are noteworthy because they are absent from the repertoire of text-based materials. 
Gr^hical depictions are important because they bear resemblance to the type of dynamic analogical 
representations called mental models (Centner & Centner, 1983; Johnson-Laird, 1983). One 
expression of testing that relies on graphical representations is the figural response item format. 
Figural response items are constrained to be tasks in which (a) an examinee demonstrates 
proficiency by adding to, arranging, or otherwise modifying figural material, and in which (b) the 
response is mentally constructed, not chosen from a list. 

Figural response items have been developed for p^er-and-pencil assessment in science 
(Martinez, 1991), as well as computer-delivered assessment in architecture, cell/molecular biology, 
and engineering (Martinez &, Katz, 1996). In these projects, item development proceeded in an 
exploratory and creative fashion, often by recruiting domain experts to construct tasks compatible 
with constraints (a) and (b). One persistent problem, however, was understanding the range and 
kinds of tasks which can be set within a figural medium, as well as the cognitive demands of those 
tasks. The need for a taxonomy of item types led to the scheme proposed in Table 1. 



Table 1 About Here 



Each of approximately 70 items was clustered provisionally with similar items. When no 
existing category could accommodate an item, a new category was created; ambiguities led to 
further definition of item types. This process proceeded iteratively, until all items could be placed 
into groups unambiguously. On the basis of iterative conceptual refinement. Table 1 was revised 
to its present form. The resulting item types are defined to be distinguishable on the basis of (a) 
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the nature of the task, (b) the given information in the problem, and (c) the cognitive processes 
used to construct a correct response. These features correspond to columns in Table 1. The last 
column lists non-exhaustive criteria that could be used to score responses. 

Each item type is now described in turn. 

TYPE I: POINT TO IMAGE ELEMENT 

The first item type involves “pointing to” named structures. In its most basic form, Type I is 
essentially a multiple-choice question but with large numbers of response options. For example, if 
a user is presented with an unlabeled map of the world, one could ask, “Where is Australia?” A 
response is indicated by moving a pointer to a specific area and marking it. Asking, “Where is 
Tongatapu?” makes the question more difficult, and if international boundary lines are removed, a 
mental construction of a response is required. Likewise, in viewing X-rays or bubble chamber 
tracks, the specialist can perceptually isolate a named element fiom a complex image. Perceptual 
isolation in a complex visual field is an important aspect of expertise in some fields (deGroot, 

1965) — and is not assessed well by traditional tests where potential responses must be marked 
clearly, pre-empting perceptual isolation on the part of the examinee. 

TYPE H: EVALUATE IMAGE 

Type n involves identifying an element of a figure with its name or symbol, detecting a structural 
error, or computing a value or attribute. For example, one could present architects with truss 
diagrams and ask them to label each truss member as being in tension, compression, or having no 
net internal force. This task might be seen as a set of multiple-choice questions, since only three 
options exist for each truss member, but in fact the problem is likely to invite complex reasoning 
because evaluating the forces within a truss member entails consideration of the truss as a whole, 
and requires inferences about effects that propagate within the structure. The evaluation of an 
image might be combined with “pointing to” image elements (Type I). For example, a technician 
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might be asked to evaluate an X-ray film by pointing to anomalies. In this case, evaluation is 
combined with the tasks of percepmally sorting and interpreting the pieces of the complex image. 

TYPE m: SHOW DIRECnON/PATH 

This category asks an examinee to show the directions of motion Ginear or curvilinear), vector 
quantities, or paths. The ability to indicate directions and paths are important aspects of 
understanding in many scientific, technical, and vocational fields. This mode allows an examinee 
to demonstrate, for example, knowledge of kinematics, including naive conceptions (Clement, 
1982). Paths of fluids, light, and electric current could be shown as well. A complex path could 
also be indicated. For example, an examinee might trace a path on a city map to show the most 
direct route from one location to another. 

TYPE IV: SHOW HEIGHT, EXTENT, OR BOUNDARIES 

Type rv items call for an indication of unidimensional extent or spatial extent. Some items asked 
examinees to indicate the height of a mercury column in a thermometer or of a liquid in a test tube. 
Architecture items involved drawing property lines while others required demarcation of building 
sections that would move independently during an earthquake. To some degree, all Type IV items 
call forjudging “how much.” The item type does not involve pointing to a pre-existing structure, 
but marking an image to indicate extent 

TYPE V: ASSEMBLE ELEMENTS 

Type V items are arguably best suited to the capabilities of a computer-based figural format. 

Within the scope of this item type is a wide range of possible items that would be impossible to 
duplicate on paper. In architecture, some of the items involved constructing a structure from a “kit 
of parts.” In engineering also, a circuit can be assembled fipom components. Other architecture 
items involved placement of pre-drawn buildings and vegetation on a site to satisfy constraints that 
are either explicit or assumed from convention. Assembly usually involves rotation and movement 
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of existing elements, but new structures could also be created and added to an assembly. Some 
architecture items, for example, asked for the placement of a water vapor barrier in a building 
cross-section. The vapor barrier would be drawn using a line-drawing tool. 

TYPE VI: INDICATE CATEGORICAL, ORDINAL, OR FUNCTIONAL RELATIONSHIPS 
These items ask examinees to depict relationships among discrete elements. Nominal (categorical) 
relationships can be indicated by separating elements (pictures or words) into lists or cells of a 
matrix. Ordinality could likewise be shown by arranging elements within pre-set or user-specified 
dimensions. For example, names or shapes of countries could be placed in a two-dimensional grid 
labeled per capita GDP and birth rate. If exact placement on these dimensions were desired, rather 
than merely ordered placement, that requirement could be specified in instructions and the response 
scored accordingly. Functional relationships could also be specified by using flow charts, 
hierarchies, semantic nets, and concept maps (Lambiotte, Dansereau, Cross, & Reynolds, 1989). 



TYPE VH: INDICATE CONTINUOUS RELATIONSHIPS 

Finally, continuous relationships can be indicated using line graphs. One science item, for 
example, asked the learner to draw the relationship between the temperature of water and heat input 
as the water changes phase from ice to liquid to steam. Continuous relationships can be scored 
according to a precise ideal response (with specified tolerances for error) or according to their 
general form. Many core concepts in science and technology can be depicted as relationships 
between continuous variables, including predator-prey relationships, supply and demand curves, 
and radioactive decay. 

Discussion 

The scheme described in this paper is not a definitive catalogue of task types possible within a 
graphical environment. It is, however, a starting point for describing existing figure-based tasks 
and test items. It is also a fnunework for item construction and for invention of new tasks 
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involving the manipulation of graphics. The importance of the scheme goes beyond the context of 
computer-assisted testing to virtually any situation where a computer is used to promote learning. 

A few generalizations can be made from the typology. First, the range of figural response 
items includes types that are especially suited to computer-based environments and difficult or 
impossible to imitate using the multiple-choice or paper-based formats. Type V (assembly of 
elements) problems are the best example. If these were reconfigured to be multiple-choice 
questions or forced to expression on paper, their cognitive requirements would change drastically. 
Similarly, for Type I (point to image element) items, making structures explicit for a multiple- 
choice version would circumvent the very perceptual isolation of elements desired from examinees. 
Multiple-choice forms of Types VI (indicate categorical, ordinal, or functional relationships) and 
Vn (indicate continuous relationships) would present the work of indicating relationships in 
complete form, leaving the user the unexciting task of selecting a correct arrangement from a small 
set of mostly-incoirect graphics. 

The focus here has been the demonstration of proficiency in a two-dimensional medium. 
However, all of the task types described could be posed in three dimensions. In some cases, such 
as assembly, a third dimension would lend greater authenticity to many problems. In other cases, 
such as graph construction, an extra dimension would add to the complexity of the representation 
and presumably to the sophistication of the knowledge and reasoning required by the user to 
respond correctly. But even a two-dimensional graphical medium can expand the repertoire of item 
types and the kind of cognitive assessments elicited by assessments. On the other hand, 
demonstrations of proficiency in a two-dimensional graphical medium might merely provide 
examinees with an alternative means to demonstrate their understanding, but showing it another 
way can sometimes be the best evidence that one truly understands (Sigel, 1991). 
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Table 1 

Taxonomy of Items for the Figural Response Format 



Type Task 


Givens 


Cognitive Processes 


Scoring Criteria 


I 


Point to image element 


Name of element 
and complex image 


RecognizeAecall image; 
possibly isolate image 
element perceptually 


Location within 
response field 


n 


Evaluate image 


Image 


Retrieve name, symbol; 
compute value^attribute; 
detect error 


Location within 
response field; 
selection/construction 
of evaluation; etc. 


m 


Show dircction^ath 
of motion 


Initial trajectory; 
vectors; path criteria 


Determine direction 
or path 


Match with ideal 
direction or path 


IV 


Show height, extent, 
or boundaries 


Referenced background 
(e.g., scale) 


Determine how much, 
where, and in what 
position 


Location within 
response field or 
position relative to 
other marks 


V 


Assemble elements; 
possibly create elements 


Element set and/or 
a partial structure 


Determine stmcture 


Location, orientation. 


VI 


Indicate categorical, 
ordinal, or functional 
relationships 


Concepts and 
group names, 
dimension axes, 
matrix, or links 


Determine categorical 
identities, orders, 
or relationships 


Location; relation 


vn 


Indicate continuous 
relationships 


Data or concepts and 
dimension axes 


Determine continuous 
relationships 
linear 
non-linear 


Location, orientation, 
straight or relation 
free-form 
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